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The Problem 

The World Wide Web has increasingly become a research hot topic, and studies about 
searching the Web are blooming across a variety of fields such as library and information sciences 
(Jayne & Meer, 1997), educational psychology (Windschitl, 1998), and business (Urgo, 1997). Yet, the 
majority of these studies focus on the technological and commercial aspects of the Web. They critically 
evaluate different search engines (Poulter, 1997) or describe specialized databases (Kurkul, 1997; Shue 
& Hue, 1997). What are missing are more empirical in-depth descriptive studies that focus on the user's 
knowledge base and cognitive and affective processes while searching the Web (Hill and Hannafin, 
1997; Kafai & Bates, 1997). 

Our interest in the Web stems from the fact that it has the potential to be a powerful educational 
tool (Hackbarth,1997; Owston, 1997; Ryder & Hughes, 1997; Windschitl, 1998). However, searching 
for information on the Web can be problematic due to the nature and vastness of its database. 

Immediate access to a large quantity of information that has not been systematically evaluated might 
present more complex problems than conducting a search in a traditional library (Fung, 1997). 
Therefore, when searching the Web, users must apply their critical thinking skills, metacognitive and 
navigational strategies in order to successfully evaluate the reliability and validity of the sources, as 
well as to avoid getting lost in cyberspace (Jones, 1997; Marchionini, 1997). 

The purpose of this exploratory study was to compare three experts and three novices in a Web 
based problem-solving context. The task was to search and evaluate Web sources so as to write a 
research paper on a topic (inquiry based instruction or IBI) that was unfamiliar to the users. This study 
differs from other studies on problem solving because the task is open-ended and it also includes 
problem finding (Getzel & Csikszentmihalyi, 1975) and as such, it provides an added dimension to this 
area of research. Focusing on how novices compare to expert Web users in searching the Web, the 
following research questions guided this study: 1) what type of navigational strategies do experts and 
novices employ to search for information on the Web? 2) what are the novices' and experts attitudes 
and feelings during the search? and 3) what type of metacognitive strategies do experts and novices use 
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during the Web search? Lessons learned from studying novice and expert Web-users, have significant 
instructional implications for educators and designers. 



Theoretical Framework 

Due to the ill-defined problem solving on the Web and the hypertext nature of the Web 
environment, theories about cognitive flexibility (Spiro, Feltovich, Jacobson & Coulson, 1991) and 
expertise (Ericcson & Lehman, 1996) form the framework for this study. From the information science 
literature, this study borrows the concept of Personal Information Infrastructure, or PII (Marchionini, 
1995) defined as 



"...a collection of interacting mental models for specific information systems; 
mental models for events, experiences and domains of knowledge; general 
cognitive skills (e.g. inferencing, recognizing salience) and specific cognitive 
skills related to organizing and accessing information (e.g., filing rules, 
reading); material resources such as information systems, money and time; 
metacognitive resources for planning and monitoring thought and action; and 
attitudes toward information seeking and knowledge acquisition... The level of 
development of a person's information infrastructure is roughly analogous to 
the level of his or her information literacy" (p. 1 1-12). 

In agreement with Marchionini (1995), in order to investigate information searching it is 
important to use the PII model, which integrates different components such as cognitive, metacognitive 
skills, attitudes and feelings as well as material resources. In this paper, only parts of the results in three 
categories (navigation, metacognition and affect) are reported. In the future, this will be complemented 
with other categories such as evaluation and cognitive strategies. 

Expert-novice differences have been studied to determine the initial state of the learner 
(novice), and what is required to become an expert. From this research it is known that the course of 
knowledge acquisition proceeds from a declarative to a procedural, condition-action form (Glaser & 
Bassock, 1989). This difference is between knowing what and knowing how. Novices may acquire the 
same amount of information as experts without knowing the condition of its application. Experts, on 
the other hand, seem to acquire knowledge in relation to its function and applicability (Chi and Glaser, 
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1985). Novices tend to categorize problems by surface features whereas experts look for patterns and 
analytic strategies (Chi, Feltovich and Glaser, 1981). Experts have more in-depth prior knowledge and, 
through chunking, can switch faster from one strategy to another when faced with difficulty on a 
problem (Presley and McCormick, 1995). Ericsson, Krampe, & Tech-Romer (1993) argue that it takes 
10,000 hours or 10 years in the field to become an expert. Due to the relative novelty of the Web, in 
this study, expertise is defined in terms of hours of Web access and not necessarily years of familiarity 
with the Web. 



Methodology 

Due to the ill-structured nature of the problem, cognitive task analysis (Shute, 1997) and verbal 
protocols (Ericsson & Simon, 1993) were used to gather and analyze information. The sample 
consisted of three novices and three experts, all graduate students at the Faculty of Education (McGill 
University). We assumed that these students from this Faculty would be better equipped to verbalize 
their knowledge. Experts were Web-based course designers and teachers. Novices shared similar 
computer literacy and research assistantship background. The criterion, which differentiated novices 
from experts, was the amount of time spent on the Web. The novices’ time was on average less than 3 
hours a week, whereas for experts it averaged 1 5 hours a week. Performance measures were the 
differences between how experts and novices searched the Web in order to solve the problem at hand 
(Table 1). The task lasted approximately 1 hour. One researcher reminded the participant to think aloud 
while searching the Web, the other one videotaped the sessions. This division of tasks was kept 
constant with all participants in order to increase the reliability of the procedure. A socio-demographic 
and training questionnaire was filled out before the task. After the task was completed, participants 
were asked about different topics related to Web searching. Their responses were triangulated with the 
training questionnaire and cognitive task analysis results. Member checks were done before the final 
data analysis. For data analysis, a coding scheme was created with 7 categories subdivided into 
subcategories and topics. The categories emerged from the initial open-coding analysis of the data 
(Strauss & Corbin, 1990). The intercoder reliability was approximately 90% (Miles & Huberman, 
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1994). In this paper, novices are identified as Nl, N2, and N3. Experts are identified as EA, EB, and 
EC. 

Findings by research questions To answer the research questions, experts and novice Web-users were 
compared in three categories: 1) navigational strategies; 2) metacognitive strategies; and 3) affect (see 
Table 1.) 

1. Navigational strategies: 

Navigational strategies included three subcategories: 2. 1 moves , 2.2 direction of search , and 2.3 
waiting . Overall, novices scored less than experts ( 1 1 0 vs. 1 52) due to their limited system knowledge. 

The subcategory Moves was subdivided into 9 topics such as 2. 1 . 1 backwards moves , 2. 1 .3 
accessing a search engine, 2. 1 .4 clicking, and 2. 1.9 reloading the Web page. Overall, novices had less 
moves than experts (72 vs. 87). Most of these moves included actions such as backwards moves, 
clicking, scanning, and reading. Backward moves and clicking were the largest in this subcategory. 
Within the novices, N3 was the one who had the highest number of moves, almost the double of N 1 (33 
vs. 16). Our interpretation is that having a similar lack of system knowledge, Nl and N3 had a different 
attitude toward the Web and learning. Nl was very hesitant and distrustful towards the quality of 
sources found in the Web whereas N3 had a more open attitude and she learned by trial-and-error using 
this strategy to explore the system features and reflecting on the small victories she achieved during the 
session. 

Backward moves are an essential part of navigation in a hypertext environment. Novices used 
them more frequently than experts (40% vs. 24%) and in a different fashion. Experts used backward 
moves to navigate freely within the system whereas novices were obliged to use backward moves 
mostly due to the perception of lack of other alternative routes to search or to get out of an impasse. 
This interpretation is supported by the results of category Affect, where novices were trapped in 
labyrinths more often than experts (10 vs. 0). 

Search engines : Novices scored lower than experts, that is, they used fewer search engines (5 
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vs. 7). This contrast was not only quantitative but also qualitative. Novices, with their more limited 
system knowledge and experience, had a more limited range of search engines to choose from. They 
could not access a search engine by typing its URL and often, as in the case of N3, accessed a search 
engine by chance using trial-and-error strategy. N3 accessed Yahoo indirectly by clicking on the legend 
that another user had compiled N3-E29. Furthermore, experts' choice of search engines was grounded 
on a variety of criteria such as the type of search they planned to follow (coarse vs. fine grain), the 
speed of access and response, and the organization of the search engine site. These choices were 
grounded in experience and prior system knowledge. Therefore, it is not surprising that experts used a 
higher number of search engines, covered a wider variety, and accessed them faster. For example, 
besides using Yahoo and Altavista, experts complemented their search with mega-search engines like 
Dogpile (EC-E159) or specialized ones such as an Education Search Engine (EB-E45). Being very 
familiar with their preferred search engines, experts knew their URL by heart and could access them 
directly. The total number of search engines used does not directly reflect in the scores of this 
subcategory because this information was coded in other categories such as Evaluation which is not 
presented in this paper. 

The subcategory Direction of search was divided in three topics: 2.2. 1 Key words included the 
number and the type of key words used during the search; 2.2.2 FIND feature included usage of the 
find key to look for a term in a site; and 2.2.3 Advanced search included usage of the search key, 
advanced search key, or refine key to either search for new sites or refine the present search. Overall, 
novices scored less than experts (28 vs. 34) in this subcategory. 

One of the novice expert contrasts was found in the Key words topic (2 1 vs. 14). None of the 
participants used synonyms probably due to the lack of domain knowledge. Both groups used several 
combinations of the search term, inquiry based instruction adding social studies or upper elementary. 

In addition to using common key words, each group also employed different terms. Among novices, 
the terms AERA (N 1 ) and bibliography (N2) were used. Within experts, EB added research on IBI and 
history. EA was the only one who used only one search term, inquiry based instruction and spent the 
rest of his time sifting through results. Regarding the use of quotations, N3 never used them as a way of 
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narrowing down the search, N1 and N2 used them sparingly; whereas experts always put their search 
strings in quotations. N3's limited system knowledge was also reflected in a lack of knowledge of 
where to type the key word IBI. For example, she wrongly typed it after the URL of McGill homepage 
in the address location. 

By far the largest contrast between novices and experts was seen in the usage of the find key ( 1 
vs. 1 1). N2 was the only novice who used this feature whereas all experts, especially EA used it 
frequently. This result is in line with the novices’ lack of system knowledge and lack of self-trust in 
applying what they know. 

In the advanced search topic, novices scored lower than experts (6 vs. 9). N3 and EB had the 
highest scores, which is consistent with their high degree of interactivity with the system. N3 spent a 
lot of time trying to narrow down her search because she believed that one has to read all the list of 
hits. She explored the results of her search only when she thought they were manageable, almost at the 
end of the search (episodel 17, out of total of 146 episodes). Experts, on the other hand, would read 
only the first couple of pages of hits. When they see repetitious sites, they would either modify the 
search string and/or change search engines or quit the search. 

Sites visited by the participants varied greatly. Novices visited a total of 45 sites whereas 
experts visited 85. There were 9 sites (7% of the total) that were shared by some participants and 
skipped by others. Some interesting patterns emerged out of the analysis of these sites (Table 2.). In 
terms of grouping, N3 was the only participant who did not share any of the sites visited by the others. 
Two sites were visited by all experts (Critical thinking in an on-line world and Prufrock) whereas there 
was not a single site visited by all novices In terms of evaluation, the criteria used most often by 
novices was authorship (N1-E22). Authorship (EA-E1 19) was also frequently used by experts but they 
complemented it with other criteria such as currency (EC-E137), and accuracy (EB-E176). When some 
sites were evaluated differently, the discrepancy was among experts. Furthermore, shared sites did not 
imply been equally judged. For example, Prufrock was a publisher site and EA figured "these people 
sell books and I am interested in articles" so he would not pursue it, EB was turned off by the site’s 
offer of "free catalogues" and EC was the only one who cautiously read more and thought it "could be 
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helpful". “Cow eyes dissection” was considered by EA as "... too applied and not theoretical enough" 
to warrant the visit whereas EB thought that it was "neat and he would definitely visit it". “Great Penny 
experience” was also visited by two experts and judged differently: EB disliked it, arguing that "...it did 
not seem to be research" whereas EC recognized that "it wasn’t academic but still was interesting" and 
wished to get in touch with the author. Another interesting point is that in both instances of shared sites, 
novices' evaluation was the same as experts'. For example, “iWonder”, was visited by N2 and EB. N2 
was impressed as well as EB who considered it as an "interesting place to be". “Project based learning” 
was visited by N1 and EA and both evaluated it positively because of its "similarity" to the topic of 
search. 

In sum, taking into consideration that experts used similar search engines and key words, only 2 
sites out of 130 (less than 2%) were shared by all of them and only 7 out of 130 (or 5%) sites were 
visited by at least 2 of the experts. This finding could be explained by the fact that the Web offers 
users a huge number of alternative routes to take. 

The subcategory of Waiting includes two topics: 2.3. 1 comments and actions while the user is 
waiting for the response from the system; and 2.3.2 accessing more than one search engine 
simultaneously. Overall, novices scored less than experts (10 vs. 31) in this subcategory. While 
waiting, novices were more patient and made fewer comments than experts, probably due to their 
limited system knowledge and expectations from the Web. Experts not only made more comments but 
also employed several strategies to deal with waiting such as: a) considering not waiting more than 30 
seconds, b) opening another browser while waiting for the response from the first one (EB), c) 
reloading or refreshing the page, and d) walking away to read a book, stretch or have a coffee. 

2. Metacognitive strategies 

This category included two subcategories: 4.1 reflection and 4.2 monitoring . Overall novices 
scored less than experts (33 vs. 47). Being that all the participants were graduate students, their 
metacognitive strategies were relatively high (41% and 59%). However, novices as compared to 
experts scored similarly in reflection (29 vs. 23) but lower in monitoring (4 vs. 24). 
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Reflection: This subcategory was subdivided in 5 subtopics: 4.1.1 reflecting on the structure of 
the paper, 4. 1 .2. reflecting positively on their knowledge; 4. 1 .3 reflecting negatively on their 
knowledge, 4.1.4 reflecting on expert-novice differences', and 4.1.5 reflecting on the process of verbal 
protocols. Overall novices reflected more often than experts (29 vs. 23). 

In reflecting on the structure of the paper, since writing research papers is a common 
requirement for graduate level, it is not surprising that all participants reflected on linking search results 
with the structure of the paper (5 vs. 6). 

In reflecting positively about their knowledge, ability and interests, novices scored higher than 
experts (6 vs. 3). This difference Probably emerged from the fact that experts did not express their 
positive feelings about their own knowledge and ability because they take it for granted. N2 (E35) 
reflected on his transfer of knowledge, and N3 (E90, El 05, and El 18) openly expressed her satisfaction 
with herself every time she managed to narrow down the search through trial-and-error. “...Yes, 198 
matches, that’s good, so I’m happy, so now I realize that I was doing something wrong, I was not 
clicking on every single subject, so now I know that the next time I do this, I have to address every 
single one” (N3-E1 18). EA, the only one who scored, reflected on his own knowledge and way of 
thinking. For example, he said: “you’ll notice that my technique in problem solving is totally 
non-linear... I think in bursts... I can think of five trains of thoughts at the same time” (EA-E128). 

In reflecting negatively about their knowledge, ability and interests, novices scored higher than 
experts (16 vs. 4 or 55% vs. 17%). This contrast could be explained by the fact that novices reflected 
frequently on their lack of knowledge about the system whereas experts reflected negatively on their 
own way of thinking. For example, N1 who scored the highest in this topic (75%) said " It's probably 
just because I don't know where to go. ..it's more a failure on my part than on how the Web is 
organized" (E 1 1 7). Among the experts, expert B, who scored the highest in this subcategory (75%), 
said " I try to keep it organized but I end up, you know, my brain goes all over the place and doesn't 
always come up the way I want it to " (El 72). Overall, novices spent 36% of their metacognitive sites 
in reflection vs. 28% for experts. 

From the participant's perspective, reflecting on expert-novice differences was not a crucial 
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topic to reflect on. In fact, both groups scored low ( 1 vs. 2). For example, N2 reflected on himself as 
being a novice (E63). EA reflected on how novices don't know what they want when searching the 
Web (El 26). 

The process of verbal protocol itself was not a crucial topic to reflect on. In fact, novices scored 
lower than experts (0 vs. 5). Being the most articulated one of the three, EA reflected spontaneously on 
his experience of using the verbal protocols method during the cognitive task analysis. For example, he 
said, “These would’ve taken about two seconds... but because I’m verbalizing it, it’s taking a bit 
longer” (EA-E26). 

Monitoring : This subcategory includes two topics: 4.2 monitoring the search and 4.2. 1 timing. Overall 
novices scored significantly lower than experts (4 vs. 24), which represents 5% and 30% respectively 
of the category metacognition. 

In monitoring the search , novices scored lower than experts (2 vs. 8). Some examples of 
keeping finger on pulse of how the search is going are as follows. N2 said "At the moment, I haven't 
learned that much... I guess a good strategy would be to get an understanding of the subject.. .let me read 
the description here" (E71). EC was aware of going off track by saying, "Let's look at Terk papers... 
now we're going a little bit away from what we're doing we're just kind of fooling around..." (E126). 

The most important novice-expert difference was found in the topic of timing , defined in terms 
of the stage of the search at which, the participant understands the meaning of the unfamiliar search 
topic or EBI. Novices scored lower than experts (2 vs. 16). This contrast was twofold, not only 
quantitative but also qualitative, that is, how many times and at what stage of the search they verbalized 
their understanding of the topic to search. N2 was the only novice who monitored the understanding of 
LBI by saying in the last quarter of the search "so again I'm realizing that a lot of the inquiry based 
learning seems to be in the domain of science education or math education" (El 13). On the other hand, 
EC in the first quarter of his search, said "...now what I found curious is that most of the information 
we're finding about EBI deals with the sciences "(E40). The conclusion here is not that the meaning of 
EBI was out of reach for novices. Rather it is assumed that their working memory was overloaded with 
figuring out how to navigate and manipulate the system. Within experts, an interesting contrast 
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emerged. EA scored the highest (10), which is consistent with his profile as an articulate and reflective 
person, who planned and implemented a coarse grain analysis focused on understanding the meaning of 
the topic to search. Halfway through his search, EA said "...so I am reconstructing my idea of what 
inquiry based instruction includes. ..student-centered, student directed, teachers as pure mentors.. .so 
now 1 have some starting points anyway, before I was totally in the dark!" (El 03). On the other hand, 
EB, despite having the highest number of episodes (217), using the highest number of search engines 
(5) and having two browsers opened simultaneously, was more tentative in grasping the meaning of the 
search topic. EB recognized that: "I am not a hundred percent sure but I have a pretty good idea.. .like 
problem solving but not quite problem solving.. .may be that maybe that’s part of my problem..." (E79). 

3. Affect 

The category Affect was subdivided into five subcategories: 3.1 labyrinth , 3.2 unfulfilled 
ex pectations , and three types of feelings or attitudes during the search: 3.3 neutral , 3.3. 1 positive and 
3.3.2 negative . Regarding these three types of feelings, it is important to note that it is difficult to 
isolate them because they are interwoven with other categories, especially with a) positive and negative 
reflections on one's knowledge (4. 1 .2 and 4. 1 .3) which was already analyzed on in this paper, and with 
b) participants' satisfaction about intermediate and final results (1.5. and 1.6) that will be reported in a 
subsequent paper. 

What really differentiated novices from experts is represented in the subcategory labyrinth . 
Novices were caught in labyrinths 10 times whereas experts never crossed that path. For example, 
among novices, N3 was the one who was caught most often in labyrinths (6 times). She started early 
during the search (E4) where she said "...just trying to figure out where I am..." Later on she added: 
"...I'm stuck, I don't know what to do next..." (E26). 

During the search, novices and experts experienced unfulfilled expectations in a similar way ( 1 1 
vs. 10). Within novices the scores in this subcategory were evenly distributed whereas among experts, 
70% of the unfulfilled expectations were concentrated on EB. Three types of unfulfilled expectations 
were identified: The first was triggered by lack of specific features in a site like dead links (N2-E79), 
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no search keys within a site (EA-E56; EB-E83), and restricted access to sources in a hypertext structure 
(EB-85). Both novices and experts experienced this type of unfulfilled expectation, which was an 
immediate affective reaction to not being able to manipulate the system as expected. Being in the 
wrong location triggered the second type of unfulfilled expectation. This type was typical of novices. 
Within novices, N3 was the only one who verbalized her disappointment of repeatedly finding herself 
in the wrong spot, namely in the neighbourhood of math instead of social sciences (E95, El 13, El 25). 
Finally, wrong assumptions about the search topic triggered unfulfilled expectations. This type of 
unfulfilled expectation was more important than the others because it had a long-term impact by 
directly steering the direction of the search. EB erroneously associated inquiry based instruction with 
instructional technology that resulted in changing the direction of his search and finding irrelevant 
sites. For example, he was side-tracked by the idea that the University of Wisconsin, which he 
perceived as having a good reputation in instructional technology, could provide a good point of entry 
into research on IBI (EB-E108). 

Novices scored lower than experts (3 vs. 5) in verbalizing their neutral feelings . The interesting 
point is that each group had different neutral feelings. N3 was the only novice who scored in this topic. 
She expressed being surprised that a course outline on the Web could present so much detail (E-136) or 
being puzzled at " finding the same search already written there..." (E-66) or being curious about some 
new features to explore (El 42). Feelings of surprise, puzzlement and curiosity found in N3 contrasted 
with experts' feelings of being in control and not getting personally involved with the search results, 
which reflects the experts familiarity with the medium and the task at hand. This attitude was reflected 
in feeling at ease with the technology (EB-E1) and indifferent about not having found the right point of 
entry expressed as "I'm not frustrated. I'm also not optimistic. I'm just indifferent because this happens 
all the time. You have to try different points of entry" (EA-E35). Self-regulation practiced by experts 
avoid "frustration and circular problem solving... people get headaches... people complain of you 
know... this fuzzy feeling in their head that they can't describe and I just I don't even get involved with 
that. If I don't find what I don't want eh. ..then I just walk away from it" (EA-183). 
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Novices scored higher than experts ( 1 vs. 0) in expressing their positive feelings during the 
search. This low score in both groups might be due to a methodological issue. Those positive feelings 



the evaluation category that will be reported in a subsequent paper. The only example for N2 of 
positive feelings about the system is "Ok, Ok. So this seems to be working. Good, so we've got 
somewhere. (E-21). 

Novices scored lower than experts (3 vs. 7) in expressing their negative feelings during the 
search (3.3.2). This, at first could be surprising. However, our interpretation is that part of novices' 
negative feelings were coded either in the subcategory 4.1.3, negative reflections on one’s knowledge 
and ability, that was already analyzed in this paper, or in the subcategory 1.5 and 1.6 that reflected their 
in satisfaction about intermediate and final results that will be reported in a subsequent paper. Novices’ 
negative feelings reflected anger about a feature of the system or feeling helpless for not knowing how 
to get out of a labyrinth. For example, on two occasions N2 said: "and again this stupid menu thing is 
kind of annoying..." (N2-E38 & E-97). N3 said "I feel kind of silly because I couldn't get out of that 
first area... I feel little yeah I feel a little like I was at a loss because I didn't know how to get out there, 
get out of that first section until I clicked on Altavista" (N3-147). Most of the experts’ negative feelings 
(4 out of 6) dealt with their impatience during the waiting. As expressed by EB "It is a little frustrating 
having to wait" (El 17). EC described his extreme frustration in naming the WWW as "oh... yak it is 
like the worldwide wait" (El 11). Experts were more impatient with the waiting than novices. However, 
the frustration that they ventilated about the system did not impact negatively on their search. The fact 
that the experts were able to continue their search unaffected is likely due to their prior experience with 
this type of situations. Novices, were more patient with the system while waiting perhaps because 
having more limited system knowledge, their expectations about the system were lower. 

Conclusions and Discussion 

This study explored what type of affect and strategies expert and novice Web-users employed 
while searching the Web. Together with other studies, (Hill and Hannafin, 1997), our results confirm 

O 



about the results of the search during and at the end of the search were coded in another subcategory in 
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the fact that it is crucial to apply navigating and metacognitive strategies in order to retrieve and 
evaluate information on the Web. These types of strategies and the importance of investigating feelings 
and attitudes during the search are confirmed by Marchionini (1995), as they are included in his model 
of Personal Information Infrastructure. As reported by Rumpradit (1998) study, experts and novices 
differed in the use of navigational due to their varying degrees of system knowledge that influenced the 
type and application of navigational strategies. Experts had a more in-depth understanding of 
navigational strategies. This was reflected in: a) the fewer number of moves and specifically of 
backwards moves; b) the higher number of search engines used; c) the direct access to search engines; 
d) the quality of sources found, and f) the use of more efficient search strategies such as the FIND. On 
the other hand, novices who had a lower system knowledge engaged in lower level search strategies as 
reported in other similar studies (Hill & Hannafm, 1997) and studies that included both print and 
electronic sources (Hansen, 1986). The novices lacked system knowledge about alternative search 
engines to use, how to refine a search and how to efficiently browse through a source. Among our 
novices, N3 frequently refined the search in much the same way as 3rd graders using a CD-ROM, who 
instead of examining the sources, preferred to refine and pose queries to the system (Marchionini, 

1 989). In our study, EA employed only one keyword and spent the rest of his time examining the 
results of his search in a manner similar to the more advanced sixth graders using a CD-ROM. 
Therefore, we can conclude that high degree of search refinement is not a strong indicator of successful 
retrieval of information. 

None of our participants were familiar with the search topic and hence none of them used 
synonyms. In this study, EB used 7 key words whereas EA used only one. The former was the least 
successful in the search compared to the other experts. This result confirms the fact that the number of 
key words used during the search is not a strong indicator of success or failure (Hill & Hannafm 1997). 
This study also indicates that novices used backward moves more often than experts. This is confirmed 
by a similar empirical study (Rumpradit, 1998) where novices to get out of trouble used the 
backtracking facility more frequently. 
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In the subcategory monitoring strategies, the inability of novices to grasp and to verbalize the 
meaning of IBI (topic 4.2.1) confirms the findings of Hill and Hannafin's study (1997). Based on 
Ausubel (1963), these authors explain that their novices, despite being familiar with the search topic, 
had difficulties in relating new information to their prior knowledge. In the present study, in which all 
participants were unfamiliar with the search topic, the majority of novices did not integrate and 
transform their knowledge about IBI, which resulted in overlooking relevant information. This is 
reflected in two results: a) novices visited a lower number of relevant sites ( 1 8 vs. 39) and b) the low 
number of relevant sites shared with experts (Table 2). 

In terms of affect, despite being impatient towards the slow pace of downloading, experts' 
attitudes were more relaxed, confident, and satisfied with their web search results. Whereas the 
unknown territory made the novices hesitant, the experts considered obstacles found during the 
information search as business as usual. Experts remained in control of the situation by monitoring 
their search and limiting information overload by browsing only the first couple of pages of the results; 
then, after evaluating the relevance of the sites, decided on a further course of action. Novices instead 
experienced disorientation and frustration when they were trapped into labyrinths. This reaction 
corresponds to the disorientation experienced in different degree by all novices in Hill and Hannafin's 
(1997) study. At what stage does disorientation becomes debilitating and hence a barrier for learning 
and continuing the Web search? In order to answer this question more research is needed. In our study, 
a strategy N3 used to get out of the long labyrinth where she had been stuck since the beginning of the 
search, was clicking all the features on the screen. This was consistent with N3's learning style by trial 
and error. 

Marchionini (1995) states, "Electronically augmented personal information infrastructures 
affect us physically, cognitively and emotionally" (p. 16). Our experts generally did not feel physical 
discomfort after spending up to 6 or 7 hours on the Web, whereas novices usually were tired after 
approximately one hour and a half. The common thread among all novices was a certain degree of 
frustration due to their lack of knowledge about search engines, and methods to narrow down the 
search. 
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Marchionini (1995) argues that besides "amplifying and augmenting our cognitive activity, 
electronic technology affects our metacognitive activity by changing our expectations... Novices often 
expect that information obtained from a computer will be more exhaustive and more accurate" (p. 1 5). 
In the present study, this attitude was found both in experts and novices. However, the reaction to 
unfulfilled expectations caused disappointment in novices and impatience among experts. 

In this study, the participants conducted the search using browsing strategies defined as: 

"An approach to information seeking that is informal and opportunistic and 
depends heavily on the information environment... (p. 100). Browsing is 
particularly effective for information problems that are ill-defined or 
interdisciplinary and when the goal of information seeking is to gather an 
overview about a topic or to keep abreast of developments in a field" 

(Marchionini, 1995 p. 103). 

In the present study, browsing strategies matched the ill-defined nature of the search topic, and the 
participant's lack of knowledge of IBI. However, novices and experts employed browsing strategies 
differently. Novice's perceptual abilities for recognizing relevant information and deriving a definition 
of IBI were limited by the fact that part of their cognitive resources were devoted to solving major and 
minor problems in manipulating the system. Experts, who were in control of the manipulation of the 
system, could simultaneously use their cognitive resources to gain an overview of the topic in order to 
define IBI and to develop a plan to filter the information. For example, EA clearly articulated his plan 
to conduct a "...generic search to get a coarse grain analysis of the topic" in order to get a feeling of the 
topic to search (E4). Within experts we found a similar browsing style between EA and EC. They both 
articulated a plan for each step of the search. They could integrate new information and quickly obtain 
an approximate definition of the unknown topic of IBI. On the other hand, EB had a different way of 
browsing. His distinctive feature was a higher interactivity with the system. Allan Collins (1996) 
argues that high interactivity with electronic learning environments might limit thoughtfulness 
"...because things move fast, and a lack of problem finding and construction by students because 
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everything they do is responsive to some situation" (p. 352). Our hunch is that EB's high interactivity 
might have limited his ability to integrate new knowledge, thus impairing his ability to define the 
search topic and to steer the search in the right direction. Although these results were obtained from a 
small sample and should not be over generalized, they are promising and appear to be an important area 
for further research in adult literacy programs using technology (Wagner, D.A. & Venezhy, 1999). 
Future replications of this study should include a larger sample size, participants should practice 
thinking aloud before the cognitive task analysis, and pre-post measures should be used to measure 
participants' self efficacy. 

Implications 

In addition to exploring expert novice differences on the cognitive and affective dimensions of 
Web searching, this study attempts to raise critical issues on the use of the Web and to identify expert- 
novice differences that might have significant instructional implications for educators and designers. 
Some implications of our findings are: (a) critical thinking skills have to be taught in order to evaluate 
multiple sources of the Web (Rath, 1997); (b) encourage using metaphorical knowledge to map the 
problem mentally or graphically; (c) in order to avoid getting lost in cyberspace, cognitive and 
metacognitive strategies including browsing, concept maps and self-regulation (Como, 1986; Winne, 
1997) should be taught and modeled not only by teachers but also by librarians and computer savvy 
students (Warmkessel & McCade, 1997); (d) exposure to the Web should be done gradually to avoid 
tiredness and frustration; e) identifying the user's personal biases about the Web might prevent 
negative feelings and attitudes during the search f) using the Web as a research tool raises important 
issues regarding the validity and reliability of sources. For example, the exact conditions of this study 
are not easily reproduced due to the dynamic and changing nature of the Web. Change occurs not only 
in the number and type of features offered to the user but also the number and type of sources. Sites 
referenced one day could disappear the next. 
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Going beyond the fascination surrounding the use of the Web and in order to employ the Web 
to enhance knowledge construction, this study has emphasized that searching information on the Web 
is a complex phenomenon that requires developing a Personal Information Structure. This conclusion 
opens up multiple challenges for the user, the instructor, the librarian, and the researcher. Despite its 
limitations, it is hoped that this study has highlighted some issues that will contribute to the scarcity of 
descriptive classroom-based research focused on Web-based instruction. 
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APPENDIX 



Category 


N1 


N2 


N3 


Tot Nov. 


EA 


EB 


EC 


Tot Exp 


TOTAL 


1. Navigational strategies 


24 


35 


51 


110 


51 


62 


39 


152 


262 






















2.1 Moves 










1 






1 




2.1.1 Backward Moves 


8 


10 


10 


28 


9 


9 


4 


22 




2.1 .2 Ser engines 2nd time 




1 


1 


2 


1 






1 




2.1.3 Search engines 


1 


2 


2 


5 


2 


2 


3 






2.1.4 Scanning, clicking 


6 


9 


17 


32 


17 


17 


11 


45 




2.1 .5 Bookmarking 




1 




1 


1 


1 




2 




2.1.6 Open favourites’list 




















2.1.7 Copy-paste 




















2.1 .8 Direct access to site/s. engine 


1 






1 




3 


4 


7 




2.1 .9 Refine, refresh, re-load page 






3 


3 


1 




1 


2 




Total subcat. Moves 


16 


23 


33 


72 


32 


32 


23 


87 


159 






















2.2 Direction of search 




















2.2.1 Keywords 


5 


6 


10 


21 


1 


7 


6 


14 




2.2.2 Use FIND feature 




1 




1 


7 


1 


3 


11 




2.2.3 Advanced search keys 


1 


1 


4 


6 


2 


5 


2 


9 




Total subcat. Direction of search 


6 


8 


14 


28 


10 


13 


11 


34 


62 






















2.3 Waiting 




4 


4 


8 


8 


10 




18 




2.3.1 Comments & actions 


2 






2 


1 


1 


5 


7 




2.3.2 Access > 1 browsers simult. 












6 




6 




Total subcat. Waiting 


2 


4 


4 


10 


9 


17 


5 


31 


41 






















2. Affect 


4 


8 


8 


20 


18 


34 


10 


62 


82 






















3.1 Labyrinth 


2 


2 


6 


10 












3.2 Unfulfilled expectations 


4 


3 


4 


11 


1 


7 


2 


10 




3.3 Neutral feelings 






3 


3 


2 


2 


1 


5 




3.3.1 Positive feelings 




1 




1 












3.3.2 Negative feelings 




2 


1 


3 


2 


3 


2 


7 
























3. Metacognitive Strategies 


15 


12 


6 


33 


24 


16 


7 


47 


80 






















4.1 Reflection 






1 


1 




3 




3 




4.1 .1 On structure of paper 


3 


2 




5 


2 


4 




6 




4.1 .2 Positively ones’ knowl, ability 




2 


4 


6 


3 










4.1.3 Negatively ones’ knowl, ability 


12 


3 


1 


16 




3 


1 


4 




4.1.4 expert-novice differences 




1 




1 


2 






2 




4.1 ,5 Verbal Protocol process 










5 






5 






20 



Total subcategory Reflection 


15 


8 


6 


29 


12 


10 


1 


23 


52 






















4.2 Monitoring 




2 




2 


2 


5 


1 


8 




4.2.1 Timing 




2 




2 


10 


1 


5 


16 




Total subcategory Monitoring 




4 




4 


12 


6 


6 


24 


28 























Table 2. List of sites shared among some participants and missed by others 



NAMES OF 
SITES 


SHARED AMONG 
PARTICIPANTS 


MISSED BY PARTICIPANTS 




Cow eyes 
dissection 


EA and EB 


EC & all novices 






Great Penny 
experience 


EB&EC 


EA & all novices 


FHSU K-3F 


EB & EC 


EA & all novices 


NYSSI 


N2, EA, EC 


N3 & EB 


Project based 
learning 


N1 & EA 


N2, N3, EB,EC 


Prufrock 


All experts 


All novices 


Ready to leam 
overview 


EA, EC 


EB & all novices 
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